Organizations that install, upgrade, maintain, and/or customize software systems (e.g., SAP ERP, or Oracle EBS), need to run many tests in order to validate the correctness of the software systems. Due to the large scale of such software systems, the testing and validation of the systems is a laborious task, often requiring several man-years to complete. In addition, it is often difficult for testers belonging to an organization to construct or select test scenarios that are likely to be relevant to their organization and be effective for the validation task at hand.
A software system belonging to an organization may be quite complex, containing a large number of software modules, and it may also be customized specifically for the organization (e.g., due to a selection of a specific set of software module and/or organization-specific customizations to software modules). However, software systems belonging to different organizations often utilize the same or similar software modules and may involve similar customizations. For example, software systems belonging to different organizations in the same field of operations may involve similar customizations that are related to the field of operations. Additionally, software systems of different organizations may involve similar updates, such as when a new build of a generic software module is released. Thus, despite organization-specific differences in software systems of different organization, it is often the case that testers belonging to the different organizations end up running the same, or quite similar, tests on their respective systems.
Current testing approaches adopted by many organizations require each organization to devise its own testing suite. Thus, each organization needs to learn which tests are effective, which aspects of the software system should be tested, and how to do so in a cost-effective way. Gaining this knowledge may require much effort and experience; in the meantime, testing the software systems may be a less effective and prolonged process. However, were organizations able to utilize each other's testing-related knowledge, which is in a sense a wisdom of the crowd (of testers), they might be able to come up with a more effective and efficient testing plan. That being said, there are many obstacles in the way of harnessing the wisdom of the crowd when it comes to testing. For one, software systems belonging to different organizations may each be customized for their respective organizations. Thus, it is not likely that a test scenario devised for a first organization will run “as is” on a system belonging to a second organization. Additionally, test scenarios used to test a system belonging to an organization often contain proprietary data related to the organization; organizations are not likely to share their testing data if it means that in the process, their proprietary data is at risk of being leaked to an unauthorized party.